Papers with review quality
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2023.acl-long)
Copied to clipboard
| Challenge: | a program chair's job is to help reviewers and reviewers make better decisions . 80% of reviewers, reviewers voted to change the criteria for soundness and excitement . |
| Approach: | a new process for matching papers to reviewers based on keywords was proposed . the authors have also brought back miniconf and RocketChat to allow for better virtual communication . |
| Outcome: | a new process for matching papers to reviewers based on keywords allowed more fine-grained control over the paper-reviewer matches. |
Findings of the Association for Computational Linguistics: ACL 2023 (2023.findings-acl)
Copied to clipboard
| Challenge: | a program chair's job is to help reviewers and reviewers make better decisions . 80% of reviewers, reviewers voted to change the criteria for soundness and excitement . |
| Approach: | a new process for matching papers to reviewers based on keywords was proposed . the authors have also brought back miniconf and RocketChat to allow for better virtual communication . |
| Outcome: | a new process for matching papers to reviewers based on keywords allowed more fine-grained control over the paper-reviewer matches. |
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (2023.acl-short)
Copied to clipboard
| Challenge: | a program chair's job is to help reviewers and reviewers make better decisions . 80% of reviewers, reviewers voted to change the criteria for soundness and excitement . |
| Approach: | a new process for matching papers to reviewers based on keywords was proposed . the authors have also brought back miniconf and RocketChat to allow for better virtual communication . |
| Outcome: | a new process for matching papers to reviewers based on keywords allowed more fine-grained control over the paper-reviewer matches. |
Towards Reliable Paper Contributions Annotation in the ACL Rolling Review (2026.findings-acl)
Copied to clipboard
| Challenge: | Identifying the types of contributions an article makes can help readers grasp its significance. |
| Approach: | They propose to use a typology to categorize articles by their contributions to improve review quality and fairness. |
| Outcome: | The ACL Rolling Review (ARR) introduced a typology requiring authors to specify their contributions to improve review quality and fairness. |
The Double Bind: Revisiting Preprinting and Peer Review Two Years After the Removal of the ACL Anonymity Period (2026.findings-acl)
Copied to clipboard
| Challenge: | ACL removed the anonymity period for conference submissions in February 2024 . |
| Approach: | They track preprinting trends for 47k publications and analyze 1.9k peer reviews . they suggest improving visibility and investing in diversity initiatives . |
| Outcome: | The proposed anonymity period was removed in 2024, but it was ineffective for underrepresented researchers . the authors suggest addressing D&I issues rather than implementing anonymity policies. |
CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells (2025.naacl-long)
Copied to clipboard
| Challenge: | Current review comment evaluation metrics rely on comparisons with a human-written reference for a given code change (also called a diff). |
| Approach: | They propose to use a reference-free metric to measure review quality like conciseness, comprehensiveness, and relevance to compare the quality of code changes with human-written references. |
| Outcome: | The proposed metric can produce fine-grained scores that have the greatest alignment with human judgment and are more sensitive than reference-based metrics. |
Program Chairs’ Report on Peer Review at ACL 2023 (2023.acl-long)
Copied to clipboard
| Challenge: | ACL'23 makes its peer review report public and an official part of the conference proceedings. |
| Approach: | They present an analysis of the factors affecting peer review and identify the most problematic issues that the authors complained about. |
| Outcome: | The authors identified the most problematic issues and provided suggestions for the future chairs. |
PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality (2026.findings-acl)
Copied to clipboard
| Challenge: | Increasing use of large language models (LLMs) in academic review has raised concerns about quality and fairness. |
| Approach: | They propose a framework to improve the quality of LLM-generated reviews by using retrieval-augmented generation. |
| Outcome: | The proposed framework improves the human-level quality of LLM-generated reviews by adopting prompt engineering and retrieval-augmented generation. |